large data sets

A practical guide to turning multi-GB/TB data work from slow, memory-heavy GUIs into fast, streaming Bash pipelines: process compressed files, parallelize with GNU parallel, wrangle JSON/CSV via jq and Miller, build idempotent, restartable scripts, and measure bottlenecks—plus installs, real-world playbooks, and common fixes; toolkit: parallel, ripgrep, jq, Miller, csvkit, pigz, pv, datamash.